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In this work we study an adaptive step-down procedure for test- 
ing m hypotheses. It stems from the repeated use of the false discov- 
ery rate controlhng the linear step-up procedure (sometimes called 
BH), and makes use of the critical constants iq/[{m -I- 1 — i(l — q)], 
i = 1, . . . ,m. Motivated by its success as a model selection procedure, 
as well as by its asymptotic optimality, we are interested in its false 
discovery rate (FDR) controlling properties for a finite number of 
hypotheses. We prove this step-down procedure controls the FDR 
at level q for independent test statistics. We then numerically com- 
pare it with two other procedures with proven FDR control under 
independence, both in terms of power under independence and FDR 
control under positive dependence. 

1. Introduction. Multiple testing has become again a flourishing area 
of research. With new challenges stemming from very large and complex 
applied problems, with implications to seemingly unrelated areas such as 
model selection and signal denoising, it is no wonder that a large variety 
of approaches and methods, sometimes competing and other times comple- 
menting, are being developed. 

Within the family of methods whose concern is about the false discovery 
rate (FDR), we follow here a rudimentary suggestion in Benjamini, Krieger 
and Yekutieli [4] (hereafter referred to as BKY) , for testing m null hypothe- 
ses Hi, i = 1, . . . ,m, using a p- value pi available for each Hi. We make use 
of the constants 

(1.1) ai = iq/{m + l - i{l - q)), i = l,...,m, 

in a step-down manner. 
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More formally, we define the multiple-stage adaptive step-down procedure 
as follows: after sorting the p- values as < • • • < P(m)) with //(j) being the 
null hypothesis corresponding to , we let 

k = max{l <i< m-.p^^j-^ < aj,j = 1, . . . ,i} 

and reject the k hypotheses associated with • ■ • ,P(fc), when such a k 
exists; otherwise, reject no hypothesis. 

Motivated by the success of this procedure as a model selection procedure 
[2], as well as by its asymptotic optimality in a mixture model noted in [7], 
we consider investigating its FDR controlling properties. To that end, we 
first theoretically establish its FDR control under independence by proving 
the following slightly more general theorem. 

Theorem 1.1. For independent test statistics, with the p-values dis- 
tributed as U{0, 1) under the corresponding null hypotheses, the step-down 
procedure with the critical values ai < ■ ■ ■ < am satisfying ai/{l — ai) < 
iqj (m -|- 1 — i), i = 1, . . . ,m, controls the FDR at q. 

We then numerically compare its power to other compatible procedures 
with proven FDR control under independence. We similarly examine its FDR 
controlling behavior under positive dependence, and find it to control the 
FDR mostly below the desired level, but sometimes slightly above it. The 
possibility of choosing different sets of constants that satisfy the condition 
of Theorem 1.1 and further offer proven FDR control under positive depen- 
dence is discussed. One such procedure is suggested, but its proven FDR 
under positive dependency comes at the expense of being too conservative. 

2. Background. This work is within the FDR framework, where the in- 
terest is in procedures that control the expected proportion of true null 
hypotheses rejected (V) out of the total number of null hypotheses rejected 
(-R), in the form of FDR = E[(y/R)I^jiyQ-j]. To understand the rationale be- 
hind the development of an FDR controlling adaptive procedure, recall first 
that when testing m hypotheses, the linear step-up procedure at level q in 
Benjamini and Hochberg [3], also called the BH procedure, runs as follows: 
let -ff(j) correspond to the ordered p- value , as before, and 

k = max{l <i< m:p(^i-^ < iq/m}. 

If such a k exists, reject the k hypotheses associated with ■ . ■ ,P{k)] oth- 
erwise reject none. When rriQ of the hypotheses are true, the BH procedure 
controls the FDR at qmo/m when the test statistics are independent. It 
is too conservative when mo/m is small. Knowledge of mo can therefore be 
very useful to improve upon the performance of the BH procedure. If mo was 
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given to us by an "oracle," the linear step-up procedure with = qm/niQ 
would control the FDR at precisely the desired level q. In practice mo is not 
known, and so the field is open for efforts to construct adaptive procedures 
that estimate this factor first and make use of it in the BH procedure. 

In BKY, a two-stage adaptive BH procedure is given based on this ratio- 
nale. It estimates mg at the first stage based on the number of null hypothe- 
ses rejected by the BH procedure, then uses this estimate to modify the BH 
procedure at the second stage. They prove that this two-stage BH procedure 
controls the FDR under independence. They also offer two multiple-stage 
versions of this, where the BH procedure is used in a similar way up to m 
times rather than merely twice. The first procedure runs as follows: let 

k = max{l <i<m:yj<i,3 l> j so that p(^) > Iq/ (m -|- 1 — j(l — q))}. 

If such a k exists, reject the k hypotheses associated with P[i), ■ ■ ■ ,P(k)] 
otherwise reject no hypothesis. A simpler version of the above procedure 
requires I = j, resulting in a multiple-stage step-down procedure with the 
constants {ai} in (1.1). Both of these procedures compare the ordered p- 
values to the same constants, and BKY offer no analytical proof of the FDR 
controlling properties of either procedure. In the first one, a combination 
of step-up and step-down methods is used. In the second simpler one, a 
pure step-down method is used. It is this second multiple-stage adaptive 
step-down procedure that we revisit in this article, offering an analytical 
proof of its FDR control under independence. It is interesting to note that 
the FDR is not controlled by the corresponding step-up procedure based on 
these constants. As the last constant in (1.1) approaches 1 as m increases, 
almost all tested hypotheses will be rejected at the first stage in this step-up 
procedure, and thus the FDR will be approximately mo/m. 

Following the work of Abramovich, Benjamini, Donoho and Johnstone 
[1], who showed the asymptotic advantage of using BH-based penalty func- 
tions for model selection in sparse situations, Benjamini and Gavrilov [2] 
investigated via finite sample simulations the performance of model selec- 
tion procedures based on the FDR control over a wide range of configura- 
tions. Their analysis revealed excellent performance for the model selection 
method based on the constants {ai} in (1.1). Moreover, the method can be 
implemented as a forward selection penalized least squares, just as the BH 
procedure can be implemented within a backward elimination procedure. In 
view of these results Benjamini and Gavrilov [2] were able to prove the FDR 
controlling properties of the multiple-stage step-down procedure, but only 
in an asymptotic setting. 

Independently, within the same asymptotic framework of a mixture model, 
Finner, Dickhaus and Roters [7] were interested in the form of an asymp- 
totically optimal curve to which the empirical distribution of the p-values 
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should be compared. They arrived at a curve of constants to which the in- 
verse of the empirical distribution is to be compared that asymptotically 
takes the form f{t) =tq/[l — t{l — q)]. As = /(i/(m + 1)), the equivalence 
of the curve in [7] and the curve to which the constants of the multiple- 
stage adaptive step-down procedure converge as m increases, is clear. More- 
over, when searching for a finite procedure, they looked at methods that 
use the constants Ui^m = iq/{m + — ^(1 — q)), with Pm/m — > 0. The Oi 
in (1.1) corresponds to = 1 in these constants. They were also able to 
prove that a few variations of this procedure control asymptotically the 
FDR. They nevertheless noted that getting a proof for the finite case of the 
step-down procedure that uses these constants seems to be a very difficult 
task. 

This brings us to the result presented in this paper in which we prove that 
for any finite number of hypotheses, tested using independent test statistics, 
the multiple-stage adaptive step-down procedure controls the FDR. 

The present adaptive procedure should be compared with two other adap- 
tive procedures, as they too have been proven to offer FDR control when 
the test statistics are independent. These are the two-stage adaptive BH 
procedure in [4] and the modified version of Storey's procedure [9] presented 
in [10]. The two-stage BH procedure runs formally as follows: at the first 
stage apply the linear step-up procedure at level q' = q/{l + q). Let ri be 
the number of rejected hypotheses. If ri = reject no hypotheses and stop; 
if ri = m reject all m hypotheses and stop. Otherwise, let mo = m — ri, 
and use the linear step-up procedure at the second stage with q* = q'm/mQ. 
The modified Storey's procedure runs as follows: let r(A) = ]^{pi < A}, then 
estimate mo by mo = (m + 1 — r(A))/(l — A), and use the linear step-up 
procedure with q* = qm/rriQ with further requiring for selection that Pi< X. 
For estimating mo, Storey [9] and Storey and Tibshirani [11] recommended 
use of A = 0.5. 

3. Proof of Theorem 1.1. In this section, we provide a proof of Theo- 
rem 1.1. Let us now introduce the more explicit notation Pi;m = P(i), so 
-fi:n ^ ■ ■ ■ ^ Pm:m are the m ordered random p- values, and {Hi,i G Iq} is 
the set of true null hypotheses. Then, from Sarkar [8] we see that the FDR 
of the step-down procedure in the theorem is given by 



FDR 



T.T.-MPl:m<a,,.. 



Pr:m< Ctr, Pr+1 : m > «r+l, -Pj < "r} 



m 



MPl:m<ai,. 




(3.1) 
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m 2 



^ ^ -Pr{Pi;„ < ai, . . .,Pr:m< ar,ar-l < Pi < Or} 



r 



m ^ 

- ( _-.N Pr{-pL:m <ai,...,Pr:m< Or, Pi < Or-l} 

(with Pm+i:m = 1, "o = 0, a„+i = 1 and ao/0 = 0). 

Let Pi-ni-i ^ ' ^ Prn-i-.m-i Ordered versions of the m — 1 p- 

values in the set (Pi, . . . ,Pm) \ {Pi}- Using the following result: 

{Pi : m ^ Ci , . . . , Pm : rn ^ Cj^ ■,C-j — \'^Pi'^ Cj } 
(3.2) ={Pi:m-l <Cl,...,Pj-_i^.^_i <Cj-l,Pj.^_;^ <Cj + l,..., 

: m-1 — Cm, Cj_i < Pi < Cj}, 

j = 1, . . . , m, for any increasing set of constants cq = < ci < • • • < Cm < 1 
(see, e.g., [6]), we notice that 

Pr{-pL:m <ai,...,Pr:m< arjOr-l < Pi < Qr} 

(3.3) 

= Pi{pI ^^_^ < ai, . . . , PiJ; ra-l < Or-l, Or-l < Pt < Or} 

for 1 < r < m [with Pq. = 0] and 

Pr{Pl:m. < ai, • • • ,Pr:m < ar,-Pi < Qr-l} 

> Pr{Pi^ < ai, . . . , Pyl^ m-1 < C^r-1,P^ < ^r-l} 

for 2 <r <m. 

Thus, using (3.3) and (3.4) in (3.1), we have 

FDR < E E MPt2-i < «i, • • • , ^'i:?: m-i < V - ^ 



ie/o ''=1 

m. 



E E y A^: ^-1 < «1, • • • , Pll^: m-1 < «r-l, m^_l > ^r} 



iG/o r=l ~'~ 

(3.5) p,(r:;_i>a,} 



E E , 1 Pr{ A^rll < «1, . . . , i^'t?: m-l < 
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^ " E E r-T P^^^l ^ < «1 , . . . , Pit. m-l < «r-l, 

1 1 '''' ' ~r -L 

-Pj, ; 1 ^ ' Pi ^ } 

m 

= a^Pr{Pi:m <«!,.. . < "r-li^'rim > "r} 

r=l 

= a[l - Pr{Pi ■ni.<ai,---,Pm:m< am.}] 

[with -P^.'^-i = 1 in the second line]. This proves the theorem. 

Remark 3.1. Clearly, the step-down procedure with the constants Oj = 
iq/ {m + Prn — ^(1 — q))t'>- = 1, . . . , m, for any > 1, will also control the FDR 
under independence. For = 1, these constants are the largest, as well as 
the power of the corresponding procedure. Therefore in our simulation study, 
to be discussed in the next section, we have used this sequence of constants 
with = 1- 

4. Performance. We conducted a simulation study to investigate numeri- 
cally the FDR control level and the power of adaptive procedures with proven 
control of the FDR. The procedures compared are the multiple-stage step- 
down procedure, the two-stage BH procedure, the modified Storey's proce- 
dure in [10], which we call the adaptive A-based step-up procedure (hereafter 
referred to as the STS procedure), and two additional procedures that serve 
as benchmarks for comparing performance. These two additional procedures 
are the BH procedure at level q and the oracle procedure that controls the 
FDR at the exact level q but is not implementable in practice as mo is 
unknown. 

4.1. The configurations investigated. The number of tests m was set at 
m = 64, 512 and 4096, with the fraction of the true null hypotheses (mo/m) 
at 0%, 25%, 50%, 75% and 100%. The p-values were generated in the fol- 
lowing way. With Zi, . . . , Z^+i distributed independently and identically as 
A^(0, 1), we first let Yi = y^Z^+i + \/l — pZi + Hi, i = I, . . . ,m, and then 
defined the corresponding p- values as pi = 1 — ^{Yi) , i = 1, . . . ,m, where <I> 
is the cdf of A^(0, 1). The values of pi were either or an equal proportion of 
1, 2, 3 and 4; the values of p were 0, 0.2, 0.5 and 0.8. The simulation results 
are based on 5000 replications. 

4.2. Independent test statistics. The results of the FDR control under 
independence for all the procedures are given in Table 1. As expected from 
the theory, the FDR level of the BH procedure is exactly qmo/m. The FDR 
level of all adaptive procedures is less conservative. In fact it is hoped that 
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Table 1 
Estimated FDR values with p — 



mo/?Ti = 0.25 mo/m = 0.5 mo/Tn = 0.75 mo/m = l 

m 64 512 4096 64 512 4096 64 512 4096 64 512 4096 

BH 0.013 0.013 0.013 0.025 0.025 0.025 0.036 0.038 0.037 0.049 0.048 0.048 

TS 0.023 0.023 0.022 0.034 0.034 0.034 0.039 0.041 0.041 0.045 0.045 0.045 

MS 0.026 0.026 0.026 0.036 0.036 0.036 0.040 0.043 0.043 0.048 0.047 0.047 

STS 0.040 0.039 0.039 0.046 0.046 0.046 0.047 0.049 0.049 0.049 0.048 0.048 

ORG 0.051 0.050 0.050 0.050 0.050 0.050 0.048 0.050 0.050 0.049 0.048 0.048 



The standard error of estimated FDR < 0.003 when mo/m = 1. BH is the linear step- 
up procedure, TS is the two-stage procedure in BKY, MS is the adaptive multiple-stage 
step-down procedure, STS is the modified Storey's procedure [10] and ORG is the BH 
procedure with the true mo. 

the FDR level of all adaptive procedures will be close to that of the oracle 
procedure and will not depend on mo/m. In spite of the fact that this goal 
is not entirely achieved, we see that the dependence of the FDR level on 
mo/m is moderated and no longer linear. 

Power comparisons are presented in Table 2. The power of each procedure 
is divided by the power of the oracle to yield relative efficiency. For indepen- 
dent test statistics the STS procedure is the most powerful in all simulated 
situations. When all hypotheses are false and m = 4096 the power of the 
multiple-stage procedure equals that of the STS procedure. In all other cases 
the power of the multiple-stage procedure takes the second place, sometimes 
very close to that of the STS procedure. 

4.3. Positively dependent test statistics. Figure 1 presents the FDR level 
of the studied procedures for p = 0.2, 0.5 and 0.8. Obviously the oracle and 
the BH procedures do control the FDR, even though now at a lower level 

Table 2 

Power relative to the oracle procedure for p — Q 



mo/m = mo/m = 0.25 mo/m = 0.5 mo/m = 0.75 

m 64 512 4096 64 512 4096 64 512 4096 64 512 4096 

BH 0.658 0.656 0.656 0.776 0.775 0.776 0.874 0.873 0.872 0.941 0.941 0.941 

TS 0.784 0.784 0.784 0.865 0.865 0.866 0.924 0.926 0.926 0.958 0.959 0.959 

MS 0.871 0.918 0.953 0.887 0.890 0.891 0.933 0.937 0.938 0.961 0.968 0.969 

STS 0.923 0.946 0.953 0.948 0.957 0.958 0.975 0.982 0.983 0.988 0.993 0.994 



BH is the linear step-up procedure, TS is the two-stage procedure in BKY, MS is the 
adaptive multiple-stage step-down procedure, STS is the modified Storey's procedure [10] 
and ORG is the BH procedure with the true mo. 
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The proportion of true null hypotheses out of the tested 

Fig. 1. Estimated FDR values for m = 64, 512, 4096. Legend: MS—solid line; BH — - 
dotted line; TS — dotted dashed; STS — dashed; ORG — dashed triple dotted. 

as follows from theory in [5, 8]. The two-stage procedure controls the FDR 
below the desired level q as well. 

For rriQ/m = 0.25 and 0.5 the multiple-stage procedure exceeds by little 
the FDR level for larger values of p. For other fractions of true null hypothe- 
ses the FDR level of multiple-stage procedure is below the desired q even 
for higher correlations. Interestingly, for positively dependent test statistics 
the FDR level decreases as mo/m nears 1, in contrast to the independent 
case. In order to examine more closely the maximal FDR level as a function 
of mo/m, we performed additional simulations for m = 512 and p = 0.8 that 
are presented in Figure 2. Note that the profile of FDR control as a func- 
tion of mo/m of the multiple-stage step-down procedure is similar to that 
of the oracle procedure. It can be seen that the multiple-stage procedure is 
quite flat as a function of m^/m between 0.25 and 0.5 (and gets its maximal 
FDR level when mo/m ~ 0.35). The maximal FDR level of multiple-stage 
procedure in our simulation is 0.061 with standard error 0.002. Therefore 
the FDR levels presented in Figure 1 should be close to the maximum. 
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Fig. 2. Estimated FDR values for m = 512, p = 0.8. Legend: MS — solid line; BH — dotted 
line; TS — dotted dashed; STS — dashed; ORG' — dashed triple dotted. 

The FDR level of the STS procedure in extreme cases is more than four 
times the declared q. Moreover, as the results indicate, it does not seem 
that the FDR of the STS procedure gets closer to the nominal level as m 
increases. 

5. Discussion. The multiple-stage adaptive step-down procedure shows 
good performances among the adaptive procedures with proven FDR con- 
trol so far known. It is second to the STS procedure in terms of power 
for independent test statistics. Moreover, its FDR control does not break 
down by much from the desired level for positively dependent test statis- 
tics. From our simulation study it appears that, although the FDR level of 
the multiple-stage procedure is a little higher than the desired level q for 
strongly correlated test statistics, it stays below q for small and moderate 
correlations. 

The interesting question is: For positively dependent test statistics, is 
there some bound on FDR that can then be used, or can the constants be 
changed to iq/{ni + f3m — ^(1 — q)), as indicated in [7], with a /3m > 1 that 
is either a constant or a function of m, so that the modified procedure will 
control the FDR at level q? The answer is yes. As shown in the Appendix, 
if we choose f3m = — q), the FDR is controlled for test statistics that 
are positively regression-dependent on subsets (PRDS) [5, 8]. Unfortunately, 
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this procedure is much more conservative, and as I3m/inn does not converge 
to as m increases, the asymptotic optimahty is lost. 

An important apphcation of the testing procedure studied here is using it 
as a penahzed variable selection procedure with an adaptive penalty per each 
variable that enters the model. Unlike STS and other adaptive methods that 
require all p- values to be at hand right from the beginning, the multiple-stage 
adaptive procedure can be applied on the forward path of nested models, 
using the penalty of the form cr^ X]f=i -2i_a./2' ^'^'-^ stopping at the first 
local minimum. In our previous simulations we found that the multiple- 
stage adaptive step-down procedure performs well as a variable selection 
procedure across a wide range of situations (even when the explanatory 
variables are correlated). Moreover, this procedure with the FDR level of 
0.05 shows the minimax performance among the studied procedures and 
situations. Both empirical results quest for some theoretical justifications 
that are all the more plausible because of the proven results of this paper. 

APPENDIX: POSITIVE DEPENDENCE CASE 

Assume that the Pj's are PRDS on the subset of null p- values; that is, 

(A.l) E{(j){Pi^ . . . , Pn) \Pi=u} is increasing (or decreasing) in u 

for each i € /q and any coordinatewise increasing (or decreasing) function cj) 
of (Pi,...,P^). 

Theorem A.l. When the Pi 's are PRDS, the step-down procedure with 
the critical values ai < ■ ■ ■ < a-m satisfying aj/(l — Ui) = iq/{m — i + (3m), 
i = 1, . . . ,m, controls the FDR at q, for any (3m > — q) ■ 

Proof. As argued following (3.1) in proving Theorem 1.1, we first note 
that 



MpL ^-1 < ai, . . . , Pit „_i < ar-i I Pi} 



f IiPi<ar) I{Pi<ar-l] 
\ r r — 1 



Let 

i;{P) = Pr{P|r:^-i <«!,•••, P^ZW m^l < I P}, 

so that, for each i £ Iq and 1 < r < m, the expectation under the double 
summation in the right-hand side of (A.2) can be expressed as follows: 

1 /(Pi<a,_i) 



(A.2) E 



^{Pi)I{Pr < < 



r 
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Now, note that ip{Pi) is decreasing, whereas ^ — is increasing in 

Pi. Therefore, the expectation in (A. 2) is less than or equal to 



E{,l;{P,)I{Pi<ar)} ^ 



a 



r 



1 I{Pi < a. 



r r — 1 



m < Or) 



< 



Or [ r r — 1 



r r — 1 

as a^/r is increasing in r. Thus, with niQ true nulls, we have 

moam rnoq 



(A.4) FDR<n.oE{v-^| 

r=l 

if P^>m{l-q). □ 



m 13m + mq 



< 
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